Delphi’s forecasting effort:
An Overview
Daniel J. McDonald
Department of Statistics
University of British Columbia
16 July 2021
Workflow
2. Flag and ‘correct’ other anomalies
# in the covid-19-forecast repo
zookeeper::make_state_corrector()
- This takes a “control theory” approach.
- Keep rolling measures of center and spread.
- Check if observations deviate too much.
- If yes, back-distribute the excess over some period.
- Fill in missing values.
- Handle intermittent (or regular) reporting issues (make simple predictions).
Do these randomly so that we don’t smooth the signal too much.
Forecasters
3. Run the forecaster
- Use one “forecast template” to produce both forecasts
- Basic idea is “we use Quantile Autoregressive Models with Covariates”
- Use 21 days of training data (x
n_locations)
- Plus more history to accomodate lags and so that we “see” the response
States: 0-1-2 weekly lags of deaths and 0-1-2 weekly lags of cases.
Counties:
## List of 3
## $ cases : num [1:10] 0 1 2 3 6 9 12 15 18 21
## $ fb-smoothed-hh-cli: num [1:4] 3 10 17 24
## $ dv-smoothed-cli : num [1:4] 3 10 17 24
This is really hard
- The Ensemble is only about 32% better than the baseline (averaged over all tasks)
- We’re about 22% better
- Over all time, we rank about 5 out of 65 teams
- Only 15 teams beat the baseline (of those submitting over the whole period)
- For County Cases, we are the best
- But the problem is even harder, and we barely beat the baseline
Important lessons:
Out-of-sample evaluation (with proper as-of) is huge.
Modular “forecaster template” is really helpful.
Nonstationarity is hard.
How hard is it?
On an equal footing, the best model beats the baseline by 20%. But, give the baseline 3 weeks of data, then it beats the best model by 20%.
